Momentum-Based Variance-Reduced Proximal Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization

Stochastic gradient methods (SGMs) have been extensively used for solving stochastic problems or large-scale machine learning problems. Recent works employ various techniques to improve the convergence rate of SGMs both convex and nonconvex cases. Most them require a large number samples in some all iterations improved SGMs. In this paper, we propose new SGM, named PStorm, nonsmooth With momentum-based variance reduction technique, PStorm can achieve optimal complexity result $$O(\varepsilon ^{-3})$$ produce $$\varepsilon $$ -stationary solution, if mean-squared smoothness condition holds. Different from existing methods, $${O}(\varepsilon by using only one O(1) every update. property, be applied online that favor real-time decisions based on observations. addition, problems, generalize better small-batch training than other large-batch vanilla as demonstrate sparse fully-connected neural network convolutional network.